# Milestone 1: Switch to SQLite for GTFS Feed Storage (complete: 20230803, tag: v0.1.0)
Currently upon performing a routing request, mobroute downloads &
caches the necessary Mobility DB GTFS archive, extracts the source
archive to a temporary directory, and loads the GTFS data into memory in
full. This milestone revolves around modifying this logic to instead
download and store data in a sqlite database.  And additionally,
rather then extracting (all) data for an entire feed upon executing
a routing request, SQL queries will be designed to load only the
data needed for the request. Preliminary work on SQL fetching and
loading to a DB has been done via the bootstrapped mobsql repository
(http://git.sr.ht/~mil/mobsql); this work needs followup, polishing,
and integration work so that mobsql and mobroute work well together.

Done:
  - Properly load source field in mobsql based on either mdb_source_id or
    provider/name unique rather then current stub limiting load to 1 GTFS
    provider
  - Remove all Mobility DB & HTTP GTFS feed fetching logic from mobroute
    repository (migrated to mobsql)
  - Remove all references to manual GTFS parsing (go-gtfs) within mobsql and
    complete migration to custom datastructures (mgtfs) for GTFS datatypes
  - Implement proper handling for fields marked as enum types in load 
    process for mobsql & add generated database enum constraints
  - Modify primary keys generation in mobsql schema creation to account for
    source field
  - Validate schema upon load by ensuring all required fields are marked
    appropriately in mobsql GTFS to sqlite conversion logic
  - Improve error handling & logging capability to cleanly report errors
    with source GTFS feed when imports fail
  - Switch to batch / transaction-based insertion for data loading in mobsql
    and add variable to customize insertion batch size
  - Add ability to customize location of sqlite database file used for
    mobsql
  - Switch the mobroute routing algorithm to load data directly from sqlite
    DB via mobsql rather then directly fetching & unzipping GTFS archives on
    each request
  - Add documentation to mobsql repository explaining functionality such
    that its GTFS-to-sqlite load pipeline capability could be used
    independently of mobroute
  - Create an initial tagged version of mobsql which mobroute will import /
    use as a go module


------------------------------------------------------------------------------

# Milestone 2: Transfers Table Generation & Multi-Feed Support (complete: 20231024, tag: v0.2.0)
Many feeds omit the (optional) transfers.txt table responsible for
listing valid transfers from one station to another. However in its
current state, mobroute depends on transfers.txt in order to determine
transfers, and feeds which omit transfers.txt fail to perform anything
but single trip/route routing requests. As such, this milestone focuses
on building out support for 'system-generated' transfers based on
haversine distance from one stop to another. Additionally, supporting
system-generated transfers also opens the door to multi-feed support,
wherein not just a single but multiple (2 through n) GTFS feeds are used
to generate routes. So in addition to system-generated transfer support,
this milestone also focuses on enabling the capability for using multiple
GTFS source archives in a single routing request.

Done:
  - Modify logic currently which uses GTFS regex to match against Mobility
    DB provider/name limiting routing to 1 GTFS feed to instead appropriately
    load arbitrary n GTFS feeds based on handle(s)
  - Create logic to generate & cache transfers table based on haversine
    distance
  - Modify mobsql load process to allow skipping non-required tables such
    as transfers.txt rather then erroring out
  - Add subcommand to mobroute CLI to tune maximum walkspeed/distance for
    transfers & create capability to (re)generate based on this parameter
  - Add & document pragmatic defaults for maximum walkspeed/distance
    for generated transfers & transfer generation methodology
  - Add reporting to core routing dastastructures to report source GTFS
    archive(s) used for which part(s) of route present for simpler debugging
  - Add logic to (automatically) utilize transfers table generation & caching
    when using >1 GTFS source OR when transfers.txt does not exist for
    single GTFS source
  - Add capability to use either (a) generated transfers, (b) transfers.txt
    based transfers, or both (a) and (b) as tunable / from CLI
  - Modify core routing datastructures to report transfer information based
    on walkspeed & distance (and additionally whether transfers included in
    routes were system-generated or based on transfers.txt)
  - Export transfer & trip part source information in available formatters
    so this information is available to end-users
  - Autogenerate transfers table using mobsql computed tables functionality

------------------------------------------------------------------------------

# Milestone 3: Performance Tuning (complete: 20231229, tag: v0.3.0)
Mobroute's routing algorithm performance is largely effected by 2 factors:
(a) load speed of getting the required data into memory and (b) executing
the actual routing algorithm - currently based on Dijkstra's algorithm,
once the necessary datastructures are available in memory. Milestone
1 builds out a new data storage model based on sqlite; and building on
this a number of improvements in this milestone can be made in regards
to (a). And as for (b), benchmarking and performance updates can be
made based on addressing data locality, minimizing lookups, and general
algorithm performance tuning.

Done:
  - Create logic to automatically create indexes on relevant fields in mobsql
    to optimize sqlite to memory/datastructures loading speed
  - Separate configuration as param for routing algorithm for tunables such
    as {walk speed, max trip duration, etc.}
  - Add ordering to data storage of stoptimes in memory such that routing
    algorithm's route sequential stoptime lookup is O(1)
  - Improve load performance by adding ability to load only given timeframe
    to memory (ability to set max trip duration) (e.g. stoptime pruning)
  - Improve debugging output to provide an overview of cost in elapsed time
    duration for each stage of routing algorithm pipeline
  - Add ability to limit stops & associated data loaded to memory based on
    criteria (e.g. stops pruning)
  - Add ability to customize maximum number of transfers in a trip
  - Add tooling or logging to assist in benchmark generation
  - Generate & document basic performance statistics/benchmarking for
    common routing scenarios

------------------------------------------------------------------------------

# Milestone 4: GTFS Spec & Unit Tests
This milestone revolves around bringing mobroute up to GTFS Static's
full specification & implementing unit tests for the same. In creating
the initial release of Mobroute, GTFS's specification was partially
implemented, however a number of edgecases were omitted such as: parsing
of the calendar dates schedule (e.g. for holidays), handling routes which
cross the threshold of multiple days, and handling multiple agencies
for a single GTFS feed to name a few edgecases. Many GTFS feeds fail
to load with Mobroute in it's current implementation and this needs to
be resolved; ensuring compliance with GTFS spec will ensure Mobroute
is usable in all metros (which distribute GTFS data up to spec). All
required fields should be addressed and mock-data and unit tests should
be added to the repository to ensure compliance. There are (optional
tagged) aspects of GTFS that may be omitted such as fare attributes and
frequencies; but parsing for and handling in the routing algorithm for
the tables marked as required in the GTFS spec (e.g. tables which make
up a valid GTFS archive) must be implemented.

Tasks:
  - Implement unit tests & GTFS mock framework for common scenarios
  - Add support for calendar_dates.txt to mobsql & routing algorithm
  - Add unit tests to ensure all cases work: calendar_dates.txt/no
    calendar.txt, calendar.txt/no calendar_dates.txt, both
    calendar_dates.txt/calendar.txt provided
  - Add unit tests & logic for routes.txt route_short_name / route_long_name
    handling; 1+ must be present
  - Add unit tests & logic for multiple agencies (multiple agency_id in
    agency.txt) & handle routes' route_id / agency_id unique combination
  - Add unit tests & logic for stoptimes crossing boundaries of days (e.g.
    stoptime with departure/arrival time > 24:00:00)
  - Add unit tests & logic for stoptimes timezone based on agency
    (agency_timezone), stops (stop_timezone), and inheritance from parent
    stop (stop_timezone)
  - Add unit tests & logic for stops.txt handling where stop_lat/stop_lon
    undefined (location_type=3 or location_type=4)
  - Add smoketesting capability / script for validating load & routing 
    capabilities for several common metros spanning international boundaries
    (EU, US, Asia, etc.) & inclusive of different GTFS source archive sizes
  - Implement integration testing via srht for patches sent to mailing list
    to ensure tests pass

------------------------------------------------------------------------------

# Milestone 5: Go Module + CLI/HTTP Server & Clients Implementation
This milestone will focus on 3 main tasks: a) Go Module componentization &
CLI/HTTP support rework b) Implementation with an existing routing or map
client (other then Mepo) and c) Implementation with Mepo. This milestone
is largely about focusing on integration work, exposing a public go module
for end-user consumption, as well as client integration exemplifying
how Mobroute can be accessible to those, outside of early-adopters /
developers, interested in building and working with Mobroute tooling. For
integration, Mepo being a close development is a natural target for
Mobroute support; but also Mobroute support will be added in another map
or trip planning application for Linux (such as Gnome Maps, Pure Maps,
KTrip, or similar etc.) via an added HTTP interface. Additionally this
milestone will add packaging for Alpine & NixOS.

Tasks:
  - Extract library for core application functionality into a go module &
    switch CLI to use go module
  - Add docstrings for all public functions for go module & add general
    go module package documentation
  - Deploy online public facing hosting (via go module index or otherwise)
    for generated go documentation for mobroute & mobsql
  - Implement HTTP server using go module allowing parity with CLI and add
    proper support for JSON output format
  - Ensure no global state / implement clean handling of concurrent HTTP
    requests either by blocking, queuing, denying a request, or similar
  - Add support for mobroute via HTTP server functionality to a Linux
    map application other then Mepo (e.g. for example, possibly target
    Gnome Maps, Pure Maps, KTrip, or a similar application); create PR / 
    submit patch for the chose external map application upstream and 
    document usecase(s) and information about Mobroute in PR/patch
  - Add documentation / walk through on using routing functionality with
    chosen external map application or mepo to mobroute documentation
  - Implement mepo ticket #65 to address making shellpipe_sync
    cancelable
  - Clean up gtk-server route to/from Mepo/Mobroute script interface
    integration & test on mobile device(s)
  - Tag a new version of Mepo that supports Mobroute and send release
    announcement to Mepo mailing list
  - Tag a new feature release version of Mobroute being the first release
    to the added support HTTP server which packagers can use
  - Package mobroute for Alpine & add mobroute-http (with OpenRC service)
    subpackage
  - Package mobroute for Nix & add mobroute-http (with Systemd service)
    subpackage

------------------------------------------------------------------------------

# Milestone 6: Feed Selection, Tooling, & Niceties
A number of end-user niceties & tooling will be built out in this
milestone critical to successful uptake of the project. The largest
feature to address is automatic feed selection, which will require
some experimentation; ultimately the ideal end-state would be such that
the user would not have to manually specify any upstream GTFS feed to
use, and similar to popular proprietary map applications this would be
automatically determined. Most likely some small tunable will need to be
specified (such as route type e.g. bus/subway etc.) to narrow down the
number of result GTFS feeds selected; the natural automatic criteria is
bounding box which can be fully automated; other filtering parameters will
also be addressed though in this milestone. Additionally to automatic feed
selection, a number of other smaller features will be built out such as:
a license command, ability to use a custom GTFS file (not from Mobility
DB), a cache clearing command, and a max filesize tunable.

Tasks:
  - Add clear subcommand to invalidate cache or remove GTFS source(s) from DB
    to enable refetching / updating data
  - Add license subcommand to display all licenses from upstream Mobility DB
    for GTFS sources used
  - Implement automatic feed selection based on user-given tunables (and
    add sensible defaults & document defaults)
  - Add default to automatically select appropriate GTFS feed(s) from
    Mobility DB based on bounding box
  - Add subcommand to fetch GTFS source(s) / load to DB cache without
    explicitly submitting routing/stops request etc.
  - Clean up & document existing output formats (json, mepolang, etc.) for
    all subcommands
  - Submit PR to mobility-database-catalogs generator script to add filesize
    as generated property (to be used downstream in mobroute) so that archives
    download size can be estimated prior to fetching in mobsql load pipeline
  - Add tunable to allow specifying transport type (routes.txt's route_type)
    for preference of bus vs train etc.
  - Add tunable to allow specifying maximum filesize & number of GTFS feeds
  - Add ability to override default loading / feed selection: Allow loading
    GTFS file(s) from filesystem not pulled from Mobility Database
  - Add ability to override default loading / feed selection: Allow loading
    arbitrary GTFS file(s) fetched from Mobility Database based on provider
    name/handle or mdb_source_id
  - Add documentation explaining functionality of default GTFS loading policy
    & info for using any of: 1. default (automatic) GTFS selection 2. manual
    GTFS file(s), or 3. specifying GTFS file(s) pulled from mobility DB based
    on handle